Skip to main content
Version: 6

Troubleshooting Guide

Installation Troubleshooting​

ModuleNotFoundError during installation​

Problem: When I attempt to install MoveIt Pro on Ubuntu 22.04, I see error messages in my terminal such as:

ModuleNotFoundError: No module named 'X'

For example,

ModuleNotFoundError: No module named 'typer'

Solution: Make sure you are not using a Python virtual environment (which is missing expected dependencies). Use deactivate for venv/virtualenv environments and conda deactivate for Conda environments.

build Troubleshooting​

Docker on ZFS​

Problem: When I attempt to build my user_image or user_workspace on a system using ZFS, I see error messages in my terminal suggesting that I am running out of disk space, even though I am not.

Solution: Perform the following steps to update the docker daemon to use the ZFS storage driver:

  • Stop docker: sudo system stop docker

  • Backup the docker data: sudo cp -au /var/lib/docker /var/lib/docker.bk

  • Clear out the docker data: sudo rm -rf /var/lib/docker/*

  • Find available zpools: sudo zfs list which gives bpool and rpool on my system. If you have no existing pools jump to the next step. The pool rpool has most of the disk space, so we will add a dataset to this pool.

    Adding to an existing zpool (rpool):

  • sudo zfs create rpool/docker

  • sudo zfs set mountpoint=/var/lib/docker rpool/docker

    Adding a new zpool (Skip this if you already added to an existing zpool):

  • sudo zfs create create -f zpool-docker -m /var/lib/docker /dev/xvdf /dev/xvdg (replace the /dev/ devices appropriately).

    (Optional) Restoring backed up docker data:

  • sudo cp -a /var/lib/docker.bk /var/lib/docker

    Restart docker:

  • sudo system restart docker

Temporary failure resolving Apt repositories​

Problem: When I attempt to build my user_image, I see error messages in the terminal such as:

# Ign:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
 => => # Ign:3 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease
 => => # Ign:4 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease
 => => # Ign:5 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease
 => => # Err:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
 => => #   Temporary failure resolving 'ports.ubuntu.com'
failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

Solution: If you are using a virtual machine (VM), e.g. through VMWare or Parallels, this can sometimes happen when the VM has been suspended for a long time. Try rebooting the VM and verify whether the issue persists.

Debian dependency problems​

Problem: As a Debian user migrating to MoveIt Pro 5.0.0, when I attempt to build my user_image, I see error messages in my terminal such as:

The following packages have unmet dependencies:
    moveit-pro : Depends : python3-ruamel.yaml (>= 0.17.16) but 0.16.12-2 is to be installed
                 Depends : python3-typer (>= 0.4.0-1) but 0.4.0-1~bpo11+1 is to be installed
E: Unable to correct problems, you have held broken packages.

Solution: Enable bulleseye-backports:

echo "deb http://deb.debian.org/debian bullseye-backports main contrib non-free" | sudo tee -a /etc/apt/sources.list

For more information on migrating to MoveIt Pro 5.0.0: Migration guide

run Troubleshooting​

ros2_control failed to increase socket receive buffer size​

Problem: The MoveIt Pro installer was unable to increase the kernel buffer size above the CycloneDDS SocketReceiveBufferSize minimum.

Solution: A recommended solution is to increase the Linux kernel receive buffer size as follows. You only need to do this once per login to your host machine, or you can save the settings by modifying your /etc/sysctl.conf file. This is performed automatically as long as /etc/sysctl.conf can be modified.

sudo sysctl -w net.core.rmem_max=2147483647

If you are unable to increase the kernel receive buffer size, you can reduce the minimum SocketReceiveBufferSize with a custom CycloneDDS configuration. Following our DDS configuration guide, you can specify a custom Cyclone DDS configuration file (adjust as necessary to point to your XML configuration file):

export CYCLONEDDS_URI=~/.ros/cyclonedds.xml

Using the editor of your choice, paste the following contents into your XML configuration file, reducing SocketReceiveBufferSize to match your system.

<?xml version='1.0' encoding='us-ascii'?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
    <Domain id="any">
        <General>
            <Interfaces>
                <NetworkInterface autodetermine="false" name="lo" />
            </Interfaces>
            <AllowMulticast>false</AllowMulticast>
        </General>
        <Discovery>
            <ParticipantIndex>auto</ParticipantIndex>
            <Peers>
            <Peer Address="127.0.0.1" /></Peers>
            <MaxAutoParticipantIndex>50</MaxAutoParticipantIndex>
        </Discovery>
        <Internal>
            <!-- Improves the performance of topic subscribers receiving large messages. -->
            <SocketReceiveBufferSize min="10MB" />
        </Internal>
  </Domain>
</CycloneDDS>
note

Hint: If you saw an error like: failed to increase socket receive buffer size to at least 10485760 bytes, current is 2000000 bytes, your kernel buffer size is set to 2Mb.

Unable to open display​

Problem: When attempting to run MoveIt Pro with a Gazebo simulation, I see error messages in my logs involving Unable to open display:.

Solution: Gazebo requires access to your display server to correctly visualize simulated scenes. Two common issues that could prevent the environment from accessing your display are:

Running MoveIt Pro from a CLI environment that does not have access to an X server, such as over an ssh connection.

The simplest solution to starting a Gazebo simulation is to launch it from a graphical desktop environment. However, if ssh is a requirement, ensure that the session has X forwarding enabled. Though there may be other issues with hardware acceleration depending on your machine setup.

Running as a non-UID 1000 user without permission to the X server in the graphical desktop.

By default, the MoveIt Pro application runs as a user with UID 1000. If this does not match your user's UID, then you will need to explicitly give xhost permissions to the application. At each login to your host machine, run the following to ensure the application has permissions to access display resources:

xhost +local:docker

DDS UDP communication errors​

Problem: When I attempt to run MoveIt Pro, I see error messages in my logs from MoveIt Pro and other ROS processes involving ddsi_udp_conn_write_failed.

Solution: Try restarting the ROS daemon with ros2 daemon stop, then ros2 daemon start. If that doesn't work, try rebooting.

MoveIt Studio fails to bind ports​

Problem: When I attempt to run MoveIt Pro and MoveIt Studio (the web UI), I see error messages in my logs from the web_ui container such as:

moveit_studio-web_ui-1        | 2023/09/26 16:23:26 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address already in use)
moveit_studio-web_ui-1        | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

Solution: Check for other processes using localhost port 80 and kill them:

lsof -i tcp:80
kill $(lsof -t -i:80)

MoveIt Studio does not finish loading​

Problem: When I attempt to run MoveIt Pro and MoveIt Studio (the web UI), MoveIt Studio does not finish loading. The robot or other objects are not displayed.

Solution: A previous process likely did not shut down properly. Close the terminals that were previously running MoveIt Pro with Ctrl+D. Then open new terminals and launch again.

Using MoveIt Pro and MoveIt Studio​

Unclear reason for failure of PlanMTCTask motion planning behavior​

Problem: My PlanMTCTask behavior fails with error code PLANNING_FAILED.

Solution: Activate MoveIt Studio's solution instrospector by clicking the bug icon on the PlanMTCTask node in the behavior tree to inspect each stage's solutions/lack of solutions and better understand the nature of the failure. If you are already familiar with the Rviz Motion Planning Tasks plugin you can also use it by launching the Developer RViz configuration:

moveit_pro rviz

Unclear planning scene state​

Problem: I am unclear about the the current state of the planning scene.

Solution: Launch the Developer RViz configuration and use the Planning Scene widget view the scene robot and geometry used by MoveIt:

moveit_pro rviz

Robot is stuck in collision​

Problem: The robot arm got into collision during jogging-based teleoperation and the robot cannot be moved.

Solution: To resolve the collision, use MoveIt Pro's Jog Collision Checking to carefully disable collision checking and jog the arm out of the collision. If the collision occurred during jogging-based teleoperation and collided with an object modeled in the planning scene, this can occur if the rate of arm movement is greater than the rate of collision checking. To increase the rate of collision checking, the following parameters can be tweaked in the MoveIt Servo YAML configuration file in your robot configuration package:

collision_check_rate: 10.0  # [Hz] Collision-checking can easily bog down a CPU if done too often.

You must restart MoveIt Pro for these changes to take effect.

Missing Info messages in MoveIt Studio​

Problem: When using MoveIt Studio, I am not seeing logged messages published at the Info level appear as a "Toast" message in the top right corner of MoveIt Studio:

I have a line in my custom behavior shared_resources_->logger->publishInfoMessage(...); that I expect to run but I don't see the logged message coming through in the UI.

Solution: Change the log level output in the UI by pressing the log level bell icon in the top right corner of the UI and checking the info level box.

Log level bell

Log level checkboxes

Joint values cannot be copied from joint jog panel​

Problem: When I attempt to copy joint values from the joint jog panel in MoveIt Studio, I see the following message in MoveIt Studio:

Text cannot be copied over an insecure (non-HTTPS) connection.

HTTPS Text error

Solution: If possible, access MoveIt Studio in your browser via localhost and not an IP address.

Multi-machine Solution: Since most distributed systems using MoveIt Pro are on a closed network, the web-based user interface is hosted as an insecure connection without HTTPS. Browsers may attempt to enforce various protection strategies from insecure connections, such as prevention of text copying. To enable all features of the user interface, make sure to allow insecure connections in your browser settings. When running MoveIt Pro locally, you can set Chrome to allow insecure connections from local hosts. However, on a multi-machine set-up the web UI is hosted at a separate IP address and Chrome may not be as permssive. Allow insecure connections, by clicking on the title bar and selecting "allow insecure connections".

Docker compose returns an error requiring unique items​

Problem: When I attempt to start a docker service using docker compose (for example docker compose exec dev bash), I see the following error:

validating /home/user_ws/docker-compose.yaml: services.agent_bridge.device_cgroup_rules array items[0,2] must be unique

Solution: Make sure you are using the same docker-compose.yaml as MoveIt Pro to start the service.

Details: The docker-compose.yaml in your user workspace requires a unique definition for each service the MoveIt Pro Developer Platform uses (without duplicates on later versions of MoveIt Pro and docker-compose). If you attempt to start a different service with a name that is currently in use (such as docker compose exec dev ... while MoveIt Pro is running) you will see an error like this. The moveit_pro run command loads your MoveIt Pro configuration before starting the docker services, so running docker compose without all the configuration parameters defined in your environment will result in a different service than one that was started with moveit_pro. If you redefine some parameters in the user workspace docker-compose.yaml, such as cgroup rules, an error will also occur. We recommend first removing any erroneous duplicated service definitions from the docker-compose.yaml and then utilizing moveit_pro shell or moveit_pro dev to enter a shell within the MoveIt Pro container, which can help prevent some such errors from occurring.

note

Note: If you use moveit_pro configure to change your user workspace while MoveIt Pro is running, moveit_pro shell may return this error.

The robot I see with moveit_pro rviz does not match the robot in the Visualization panel on the web interface​

Problem: Using RViz I see a different robot (or robot configuration) than the one I see in the MoveIt Pro web UI

Solution: Make sure your configuration package's config.yaml uses the same URDF as the robot_state_publusher.

Details: When setting launch_robot_state_publisher: False in config.yaml MoveIt Pro will expect the node to be launched by another process. It is the responsibility of that process to load the same URDF as the one given to MoveIt Pro. In config.yaml, ensure that the following lines point to the same URDF used by the robot_state_publisher node (note that the SRDF and URDF may be in two separate repositories if needed):

  # Parameters used to configure the robot description through XACRO.
  # A URDF and SRDF are both required.
  # [Required]
  robot_description:
    urdf:
      package: "<ROBOT_DESCRIPTION_PACKAGE>"
      path: "description/my_robot.xacro"
    srdf:
      package: "<MOVEIT_PRO_CONFIGURATION>"
      path: "config/moveit/my_robot.srdf"
    # Specify any additional parameters required for the URDF.
    # They will pe passed to the URDF xacro as arguments
    # [Optional]
    urdf_params:
      - ip_address: "0.0.0.0"
      - additional_arguments: "example_arg"

Networking and PC Setup​

Multiple computer setup cannot communicate via DDS​

Problem: In a multiple computer setup, I can ping the other computer but can't communicate with it via ros2 run demo_nodes_py talker + ros2 run demo_nodes_py listener

Solution: Check that your network interface adapter and peer address entries for your method of configuring DDS are correct as explained in the DDS configuration docs.

Multiple computer setup error messages that "Action server not available"​

Problem: Multiple computers are connected over the network, but the agent cannot seem to access action servers or services provided by the robot drivers, or discover controllers, resulting in errors such as:

[ERROR] [1658269925.062171689] [MoveGripperAction]: Error code: -25; Message: Failed to send trajectory execution action goal to server.; Details: Action server not available.
...
[ERROR] [1658269931.800549146] [Teleoperate]: Error code: 99999; Message: Failed to enable required controller servo_controller; Details: Service /ensure_controller_is_active failed with message: Failed to ensure that specified controllers are active.
...
[ERROR] [1658269939.054476603] [moveit_ros.trajectory_execution_manager]: Controller '/joint_trajectory_controller' is not known

Solution: Check that your network interface adapter and peer address entries for your method of configuring DDS are correct as explained in the DDS configuration docs.

Multiple computer setup does not have synchronized clocks​

Problem: When using multiple computer setups, the computers' clocks go out of sync. This can been seen by messages arriving from the future. Or TF complaining about old messages.

Solution: To solve this, one or both of the machines should install chrony:

sudo apt-get install chrony

If any machine in the setup does not have access to NTP, the machine that has chrony installed can serve as a NTP server. Suppose the drivers RTPC does not have an internet connection, but has a direct link to the agent PC. On the main PC (usually the PC running the Agent), install chrony and then add lines to /etc/chrony/chrony.conf to allow the drivers RTPC to connect to it. For example:

# Allow drivers PC to use the agent PC as NTP server
allow 192.168.1.2  # This should be the IP address of drivers RTPC

Then restart the service:

sudo systemctl restart chrony.service

On the drivers RTPC, reconfigure timesyncd to use the agent PC as an NTP server. Modify /etc/systemd/timesyncd.conf to have:

[Time]
# Use the Agent PC as the primary NTP source
NTP=192.168.1.1  # This should be the IP address of the agent PC

And restart the service:

systemctl restart systemd-timesyncd.service

For more information on more complex installations: Setup guide

Robot Hardware and Cameras​

Cannot connect to RealSense camera​

Problem: I cannot connect to my Realsense camera.

Solution: Reconnect the Realsense and then check if you can view it in the realsense program: if the program is not on your system, follow the Realsense Drivers instructions. The expected set of realsense packages should look something like this:

$ apt list --installed | grep realsense

librealsense2-dbg/focal,now 2.48.0-0~realsense0.4976 amd64 [installed]
librealsense2-dev/focal,now 2.48.0-0~realsense0.4976 amd64 [installed]
librealsense2-dkms/focal,now 1.3.18-0ubuntu1 all [installed]
librealsense2-gl-dbg/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-gl-dev/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-gl/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
librealsense2-net-dbg/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-net-dev/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-net/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
librealsense2-udev-rules/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
librealsense2-utils/focal,now 2.48.0-0~realsense0.4976 amd64 [installed]
librealsense2/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
ros-humble-librealsense2-dbgsym/focal 2.48.0-1focal.20210701.133935 amd64
ros-humble-librealsense2/focal,now 2.48.0-1focal.20210701.133935 amd64 [installed]
ros-humble-realsense-msgs-dbgsym/focal 2.0.8-2focal.20210630.230828 amd64
ros-humble-realsense-msgs/focal 2.0.8-2focal.20210630.230828 amd64
ros-humble-realsense2-camera-dbgsym/focal 3.2.2-1focal.20210705.140737 amd64
ros-humble-realsense2-camera-msgs-dbgsym/focal 3.2.2-1focal.20210705.135802 amd64
ros-humble-realsense2-camera-msgs/focal 3.2.2-1focal.20210705.135802 amd64
ros-humble-realsense2-camera/focal 3.2.2-1focal.20210705.140737 amd64
ros-humble-realsense2-description/focal 3.2.2-1focal.20210705.140354 amd64

Cannot launch RealSense after performing driver update​

Problem: Can't use 2 Realsense video feeds after required driver update.

Solution: Use realsense_ros - v3.2.2, librealsense2 - v2.49.00. To downgrade these packages run install_librealsense.sh in moveit_studio/bin and rebuild your workspace.

RealSense node messages that "reduced performance is expected"​

Problem: One of the cameras does not come up properly when MoveIt Pro is launched:

[realsense2_camera_node-15] [RealSenseCameraNode]: Device 123456789012 is connected using a 2.1 port. Reduced performance is expected.

Solution: 1. Verify that the cable and USB port are both rated USB 3. 2. Unplug the cable from the camera and the computer and plug it back in. 3. Open the realsense-viewer and perform a hardware reset by clicking “More” and selecting “Hardware Reset”. 4. Try connecting a different cable.

MoveIt Pro simulator's camera feeds have low framerate​

Problem: I have an Nvidia GPU and MoveIt Pro's lab_sim robot config's camera feeds are running at less than 10 FPS.

Solution: You will need to update your user workspace's Dockerfile to install Nvidia drivers inside the container. Refer to the instructions in the linked Dockerfile.